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Tnt r oduc t ion 

The following survey report was undertaken to serve as a foun- 
dation for further study and more intensive research on the evaluation 
of educational system outputs. 

It was not intended that this report should provide educational 
planners and decision-makers with a resolution of problems. Rather 
it is a discussion of the situations from which the question of system 
evaluation arises. It is also a bto'ad examination of the points of 
view which have been expressed and of the means which have been pro- 
posed and taken to deal with educational evaluation. 

The intent is to turn a page in the story of educational de- 
velopment planning by focusing upon the pressing need to know more 
about the value of efforts Which have been made and which are contin- 
ually in the process of xe- consideration for further decisions. This 
report attempts to ascertain the ; potential for further study of the 
basic problems of evaluation in education and most specifically of 
the quality and nature of the output s of school systems. 

There is continual popular demand for Change and innovation iu 
education, as if these were either magic words in themselves, or as 
if they represented processes which can be immediately applied for 
instant results. Given the current considerable demand also for more 
education and for more ’relevant’ education, it is ever more important 
that evaluation be considered as a sine qua 'non of educational plan- 
ning. The introduction of Change in itself’ is only a part of planning , 
and ‘ the incorporation, or appending, of evaluation processes to educa- 
tional procedures is therefore a basic requirement. 

Purpose of the study 

The following study will cons ider the demands for evaluation; - 
as they refer t:o L individual needs and to national r nqu iremenf s. 
will review the motivation for such demands and. the means now .in use.- 
and those proposed for educational evaluation on a system basis. Re- 
levant research and research proposals will be reviewed. 
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We will also indicate the national and regional agencies, the 
research centres, the international services, and the leading indivi- 
duals who have been working on the problems of evaluation of school 
systems. 

Finally, we will suggest areas for potentially fruitful re- 
search on this subject, noting the varieties of the studies which 
might be made and the directions which such work could take. 

The urgent interest in evaluating the results, or outputs, of 
education stems from the basic fact of the costs of education. In 
every country, developed or developing, costs for education represent 
large, if not major, portions of national budgets. The efforts and 
time of increasing numbers of people are represented both by the stu- 
dent population and by the teachers and officials' who are engaged in 
school system service* ‘ * 

"The concept of internal efficiency requires some assumptions 
about the objectives or measurable achievements of an educa- 
tional system ... Results or objectives are related to the 
measurable efforts needed to achieve them • for example, tea— 

• chers v time, students’ time, use of facilities and other re—., 
sources. These inputs* can be measured as expenditures or 
costs The expenditure or cost bf a certain course of 

learning can then be related to the attainment or objective. 11 
Economically, then, the importance of the education investment 
is significant, and is so recognized by economists in spite of the 
relative difficulty rhey have^ in determining linear connexions between 
increased investment in education and increased economic development.^ 



Friedrich; Edding , "Educational Resources and Productivity" , in 
George Z.F. Bereday, (editor) ,■ Essays on World Education; The 
Crisis of Supply and Demand , Nex* York; Oxford University ; Press, 
1969, pp> 22-23. 
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However, for national populations education and its presumed 
effectiveness promises much. There is the direct certification of 
individuals for useful and well-paying occupations. There is the 
direct relationship to social status which is recognized as achieved 
by individuals who hold certificates and degrees. There is the over- 
all tendency of an educated population to meet the requirements of 
manpower in a developing and changing economic picture. Also, a well- 
educated, broadly developing population, well-employed, provides a 
national sense of well-being and of constructive forward movement. 

A country of people who are receiving new educational opportunity and 
who find an open system is seen as striving, developing, and, perhaps, 
even succeeding* 

Yet in all systems, in all countries, there are evidences of 
lack of success. Within this context, questions begin to be raised. 
When large investments are made in education, drop-outs and repeaters 
must represent a cost. We have the suggestion of Rist that; 

"The success of an educational institution ... should not be 
measured by the treatment of the high-achieving students , but 
rather by the treatment of those not achieving. 

When certificates and degrees are awarded, each recipient, though a 
’winner', must be somewhere on a continuum from the very best level 
down io the merely passing. The question of quality arises, since 
sooner or later some use is to be made of such cert if icates and de- 
grees. Whether used in the economy, in teaching, or in Continued 
education, they presume to indicate the educational and intellectual 
achievements of their holders. 



1 Ray C . Rist, "Student Social Class and Teacher Expectations'^ 
Harvard Ed ucational Review , Vol. 40, No. 3, August 1970, p. 448. 
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Further questions arise from parents and teachers as to how 
much effort should be made to reduce drop-outs and repeaters. Pro- 
posals are made for varieties of new or re-discovered procedures in 
education; remedial courses, smaller classes, use of audio-visual 
(and tele-visual) aids, use of programmed learning devices (books 
and machines) and the relaxation of required courses so as to estab- 
lish new standards by which individual success in education can be 
determined. Current demands for relevance in curricula, for courses 
which matter ’here and now’, and for new learning modes to go with 
the realities of the 'here and now’ , are raised because there is 
general agreement that when students are motivated by the curriculum 
and by the methods of instruction they will succeed in reaching their 
educational goals. 

The formal versus the ’informal’ system 

The development of demands for educational reform has come 
principally in developed countries where the evidence of the existence 
of two, competing, educational systems has been quite strong. On the 
one hand, there is the formal structure with its sequences of levels, 
its examination system, its screening-out processes, its concentra- 
tion on measurable evidence of educational success. The rational 
system, or establishment, provides forward movement to some, elimin- 
ates others, and makes a number of choices available to still others, 
while operating on the continuing assumption that it is perpetuating 
national traditions and culture and establishing and maintaining 
ethics and morals. 

On the other hand, there is a quite ’informal’ educative appa- 
ratus making itself felt with varying effectiveness upon the popula- 
tion, particularly that portion which is of school age. We refer to 
the mass media, newspapers, magazines , books, radio, moving pictures, 
television, all of which are recognized as affecting the liyes o£; :: 'fj 
those whom they reach. Tlis is not to say that such media are totally 



O 

ERIC 



9 



IIEP/RP/8 - page 5 



effective, since if they were we would not have the evidence that 
such devices can be and are either physically turned off or emotion 
ally 1 tuned “out ' even in authoritarian States. 

Nevertheless, in countries where such media are operative an 
abundance of impressions is made in contrast perhaps with those made 
or desired by the school system as it is formally constituted. Xt 
is particularly in the broad areas of religion, sexual behaviour, 
war, and personal ethics that such divergences are most characterized. 
It is also in such contexts that the objectives of national systems 
are most tenuous, undefined, and hesitant, and where, necessarily, 
evaluation of effectiveness is least in evidence. This condition is 
likely to be as true in developing countries as it is in the developed 
ones. . . 

1 J ' 

Yet another aspect of the informal educational sy stem is that 
represented by the apparent relationship between the schopl-age popu- 
lation and. their elders. In this regard what the school system 
teaches about morality is of r?n negated in practice by the older. 
generation, more accurately designated , as the 'holders of power . 
Generalizations about human dignity are shown to be ephemeral when 
the treatment of labour, or of minority groups, or of women, is ex- 
amined. Such clear cpntrasts between reality in the behaviour of 
controlling adults and the precepts of the organized educational sys- 
tem provide a simple, if not simplistic, demonstration of the real 
purposes and effectiveness of that system. Such demonstration is, in 
effect, an evaluation based on empirical evidence. 

In all such aspects, the consequent demoralization of the . 
youth, a recurring phenomenon surely, takes on greater force since 
many of the members of the educational establishment are themselves 
coining to reject their assigned roles and to join in the clamour for 
change. 
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Leve ls of evaluation 

Thus, the problems of educational evaluation may be expressed 
on several planes. On the one hand, educational evaluation wxthxn 
an established system has tended to focus upon the achievement of 
cognitive skills by individuals. This has been a clearly-defined 
process designed to satisfy some clearly-defined goals as agreed upon 
by the educational community and its clientele. 

In this regard the efficiency of a system may be determined by 
its lessening number of drop-outs and repeaters, by its continually 
expanding avenues of access tc those desiring its services, and by 
the best return, in the above terms, to the investments which have 
been made • 

On a completely different level is the determination that edu- 
cational systems products (students) are lacking in certain charac- 
teristics, or curricular inputs, which they may be finding in the 
’informal' system. If the formal system is engaging in certain prac- 
tices which represent costs, personnel and time, such as teaching 
about ’ culture’ , the 'arts' , 'history' , Offering 'personal guidance’ , 
'health education’, and so on, these too must be subject to evalua- 
tion. Since the effort is being made, the effectiveness like that 
of reading or spelling, or knowing the dates of famous battles, must 
also be put to the test « 

On still another plane is the question of whether the educa- 
tional - isystem can be evaluated in terms of its ability to remedy or 
compensate for the debilities of the social and economic systems. 

With the growing expansion of school Systems , the entry of 'disadvantaged' 
urban children, poor rural children, ? culturally-deprived’ chxldren* 
and others lacking in educational 'readiness', has offered a challenge 
which educators world-wide have been required to face. Some have 
welcomed the challenge and sought to offer revised methods and compen- 
satory curricula, 'head starts', earlier admissions, and the inclusxon 
of indigenous (though non-professional) personnel into teachxng sxtuatxons 
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Others have faced the same challenge and have clung to the con- 
cepts that true: «mer it will come to the fore,, that equality of oppor- 
tunity once having been offered, the challenge is now to the indivi 
dual and to his family to make the most of it. Under these conditions 
the rigours of examinations and of customary evaluative procedures 
have maintained and exacerbated the eliminatory processes, the early 
cut-offs, the dumping of natural resources (human) , and the develop- 
ment of rigid class distinctions. In both kinds of responses to 
existing conditions, the problems of educational evaluation come to 
the fore . : 

The question. does remain however as to the role of education 
in making- amends for social or cultural ' slings: and- arrows’. If, as 
the answer is often made, what else is. education for then the tasks 
require that education be well-armed, well-stocked with both talent ■ 
and money, and provided with the power that comes from working on 
equal ' levels with those that hold the strings of the purse.: and the 
reins of decision. Indeed , if the answer is af f innative to the ques- 
tion about education* s role as ■ a change agent within, the social, order, 

then surely the means to : evaluate its effectiveness in snch a task 

must be sought tirelessly and objectively*..- - 

Evaluation begins with the individual. The results of indivi- 
dual testing (not teacher reporting, which is considered as top un-v 
reliable statistically) are compiled so that-, groups, of ; ttidividualSa. . i. i 
are seen as representing large-scale tendencies*. . Their grouped scores, 
become the bases by which systems are evaluated. 

Thus, when in New York City (for example) , in the 1960's, 
large groups of children showed low scores in reading, the system was 
declared to be ineffective in achieving its goals. This was without 
regard for the possible intervening elements in the social situation, 
in the state of children's health, or in the aspects of home life. 
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Xn the recent XEA studies of llusen and his associates , 
grouped scores, cross-sectional, were related (or relations were 
sought) to conditions of family life, of the educational environment, 
and of expressed attitudes and opinions. 

The difficulties are expressed succinctly by Porath who writes: 

"The final educational output is not produced only in the 

schools but also at home and elsewhere, before, during and 

-2 

after the period of formal schooling ... " 

Since the evaluation of informal system effectiveness is still 
in a Marginal and relatively undefinable condition, it would be more 
useful to focus upon the formal educational structure. Further, the 
investment of costs, personnel and time in this system are explicitly 
dedicated to the purposes of education. 

E valuative targets and decision-making 

The bulk of evaluation of system effectiveness has been in 
the achievement of individuals on set tests, with the data accumulated 
and grouped. Let us not, however, overlook those evaluation practices 
which take place in the home, in the market place, and often in this 
voting centres, when the ordinary citizen is enabled to express his 
evaluation of education by criteria of his cwn choice. 

In such circumstances, as worker, parent, employer, or general 
observer, the criteria of the common man regarding the success of his 
educational establishment are likely to be in close relation to his 
own goals and his own experiences. 



T. Husen, (editor) , International Study of Achievement in Mathematics 
Volume lr. New York: John Wiley and Sons, 1967. 

Yoram Ben-Porath, "Aggregate Costs, Output and School Achievement", 
in Donald E. Super (editor) , Toward a Cross-National Model of 
Educational Achievement in a National Economy , New York: Teachers 

College, Columbia University, 1967, p. VII-1. 
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It is not facetious to note that politicians have, per se , no 
built-in or objective criteria for determining their attitudes toward 
educational costs, changes, and development other than chose they 
have gained as perceptive laymen. Initially, therefore, social and 
economic decision-makers look at school systems through the eyes of 
the man in the street. 

Thus the view is taken that open access to education is de- 
sirable and should be available to all. It therefore follows that 
once in the system, the processes of education to which a student is 
exposed should make of him something which he formerly was not. That 
is, he should be cultured, literate, skilled, communicative, loyal to 
traditions and accepted norms, and, as important as any of these, em- 
ployable and paid on a living standard not available to one with less 
education or none. 

If this process of evaluating education is an empirical one, 
it is surely practiced by political decision-makers with a keen sense 
of constituents’ demands, by parents with a real knowledge of oppor- 
tunity costs expended (actual or with regard to young children nearing 
school age) , and xsrith most students who are mature enough to seek 
their own advantage actively. 

Thi difficulty with this form of evaluation is that it is not 
only practiced in countries which are developed and which are develop- 
ing, but it is almost universally practiced within each country. And 
it is beginning to be linked, beyond- the concepts of curricular and 
methodological relevancy, to concepts of accountability. People are 
beginning to apply their empirical concepts of educational system 
effectiveness to what they are experiencing, to what their school-age 
children are experiencing, and to what the post-school population, 
the outputs (drop-outs and put-outs) , are experiencing. With the 
focus on accountability, when all is not well, the clamour arises for 
someone or something to be looked at and studied, and for relationships 
between the educational inputs and fhe outputs to be clearly disclosed. 
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Criteria for study of evaluation 

But accountability, while a reasonable and desirable concept 
to foster, cannot be properly determined nor - and this is even more 
important in our view than accountability alone — can appropriate re 
forms and corrections be applied Unless certain more objective steps 
are taken. Evaluation must be based on specific data, quantified in- 
formation, and objective procedures. Biases, hunches, sudden en- 
lightenment must be used sparingly and, when used, carefully labelled 
in educational evaluation. We prefer, rather, to point out that our 
approach to evaluation is based on goals, processes and results. 

Ue must, therefore ask three questions: 

(1) what are the goals of the educational system which have 
been expressed, accepted as appropriate and feasible, and, 
transmuted into educational curricula? 

(2) what are the curricula which are combining content and 
methodology .and which will provide specific experiences 
for students ; so as to achieve the goals of the system? 

(3) to what extent is the measurable output of the system in 
conformity with the purposes for which the system has 
been operating? 

Subsumed within each question above are several other questions 
each of which is necessary to consider in the further development of 
educational evaluation. 

(1) Thus i with regard to goals: -h 

(a) how are school system- goals determined? 

The formulation of goals: is- proposed by Good lad an 
a very specif ic .manner: *'We need a national body of 

i,; leading citizens whose prime purpose is to give con~ 
tinued attention to the formulation of educational 
^ ■ ’ - a uns ... ,i ■ . : 1 ' 



John Goodlad , School Curriculum Reform in the tfiii ted States, 

New York: The Fund for the Advancement of Education, 1964, p. 81. 
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With somewhat different emphasis. Smith states that: 
n the problem of objectives for American schooling is 
a problem calling for highly competent professional 
resolution ... The determination of objectives that 
will give curricular and other operational meaning 
to the central purpose in the years ahead calls: for 
sophisticated theoretical, technical and administra- 
tive decisions. 

(b) to what extent do goals reflect realistic national 
needs? 

(c) how are goals translated into operational school 
processes? 

(d) what can be done to make goals explicit and amenable 
to both curricular experiences and evaluation? 

The relationship of objectives to evaluation is 
stressed by Goodlad, when he says: Heeded are 

evaluative criteria in the form of educational ob~ 
jectives which have been agreed on ... It is recoin - 
mended that curriculum investigators ... be required 
to submit statements of such objectives as well as 
plans for evaluating their attainment. These must 
not be simply statements- 6f purposes for a -given 
project but, rather , statements of the kind of be- 

ji'2 

ha.viour sought in the students." 



Philip G. Smith, "Objectives for American Education", in Stanley 
Elam and Gordon Swanson, (editors) , Educational P lanning in the 
United States , Itasca, Illinois: Peacock Publishers, 1969, p. 9. 

Goodlad , op » crt . , p . 82 . 
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( 2 ) 



Edding 



With regard to curricula: 

(a) which disciplines in the primary and secondary levels 
are directly responsive to prescribed system goals? 

(b) how can the cognitive-or iented goals be translated 
into curriculum sequence so as to provide the best 
methodological practices to varieties of students? 

(c) how can goals which are non-cognitive be achieved 

in curricular experiences which are basically re- 
quired to be explicit including: learning sequences, 

prepared syllabi, teaching aids, and trained teachers? 
Edding refers to this question; 

’'Other unsolved problems lie in the difficulties of 
measuring achievements other than knowledge and vo- 
cational skills. Yields of education rarely con- 
sidered in examinations are, for instance, knowledge 
of how to learn and eagerness to continue learning; 
ability to co-operate, to take initiative, to make 
decisions under conditions of uncertainty? and 
virtues like tolerance, honesty, self-control, and 
creativity. These are possibly the most important 
results of education, but they are harder to measure 
then the performances usually judged and marked in 
certificates."^ 

(d) how can reasonable goals which are operating in 
school curricula become feasible in countries where 
resources are scarce, teachers unavailable or un- 
trained, and the wastage rate too high to expose a 
school— age population long enough? 

(e) to what extent can curricula »teal with the large 
variety of system goals, often including skills. 



, op .cit . , p . 24 . 
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knowledge, attitudes, citizenship, personal adjust 
meat, creativity, and self-realizations, in view of 
the limited time and resources of the system? 

(3) With regard to evaluation of outputs; 

(a) how can system goals be used as criteria for evalua- 
tion of the effectiveness of system processes? 

(b) what are the most appropriate points at which to 
measure the accomplishment of students w thin a 
system? 

(c) what instruments measure student achievement, with 
validity and reliability, as a function of their 
experience within the school system? 

(d) what characteristics of students are; 

(i) measured objectively and relate to system 
effectiveness? 

(ii) not measured at all but should be itt view of 
system objectives and curricula? 

(iii) not measured and heed hot be since they do not 
have anything to do'" with" the purposes of the 
educational system? 

(a) how can various evaluation procedures of students 
during their school enrolment, at terminal points 
after each level, and at long-term, be considered 
as ' dl early indicative of system effectiveness? 

(f) how can estimates of system effectiveness be utilized 
1 ’ f or reform of curriculum* administrative practices, 

and system goals? 

The foregoing considerations have been developed out of a 
study of the problem of evaluation as it is treated both in existing 
practice and current research. As the needs of school system clien- 
tele, critics, and technical ’assisters’ become more clearly inter- 
locked, the need to approach evaluation of school systems becomes 

more urgent . 

O 




18 



IIEP/RP/8 - page 14 



It is not sufficient for empirically-drawn conclusions to 
govern the action of educational planners and decision-makers. Nor 
can cross-sectional 'cuts' of several systems provide us with more 
than 'still-life' portraits of the systems. Data obtained from such 
practices are of interest to us when they relate scores of students 
to internal system elements which may or may not represent cost in- 
puts or curricular processes. This does not, however, yield evalua- 
tion of systems since there are no criteria for evaluations, no in- 
dications of pupils' initial status and subsequent gains as a result 
of system processes, and no intra-system comparisons of reliably- 
maintained controls. 

Goal-setting and goal characteristics 

Evaluation is needed to determine how well system goals have 
been achieved. This has been stated before and is the fundamental 
basis for research in evaluation. We must, of necessity call atten- 
tion to the problem of goals . In the continuing search for criteria 
of educational effectiveness, it is reasonable to ask about the na- 
ture of the goals which are found either explicitly in educational 
systems* declarations of purpose, or implicitly in the nature of the 
evaluations which are undertaken v 

In most countries, educational goals tend to be set and are 
characterized as follows: 

(a) they represent the least common denomination of educational 
purposes, i.e. , purposes about which almost no one would 

quarrel. •' ' 

An illustrative statement by the then Swedish Minister of 
Education (4 December 1968), Mr. Palme describes his 
country's basic goals: 

"The goals cannot be different for different parts- of our 
education ... The individual is given opportunities to 
fulfil himself. Education is an absolute necessity for a 
living democracy. The goal of education should be to 



10 



IIEP/RP/8 - page 15 



create increased equality. The school should break down 
social distinctions. The school is one of the most im- 
portant means of changing society. 1 

A similar general statement is made by Smith regarding 
educational 'goals in the United States: 

"There are three sets of factors that constitute the 
basis for control over experience .. . organized human 
knowledge ... individual qualities of effective intelli- 
gence and personality , ... and the. cultural values and 

customs of our society. 'The central purpose of American 
schooling should be to teach in each of these areas, the 
skills, habit si understandings, and attitudes that are 
most educative, that is y .those which most enhance control 

and thus make possible effective choice and hence indivi- 

2 

dual and social freedom. " 

(b) they tend to be quite explicit in relation to the 

cognitive domain: the communications skills (reading, 

writing, speaking, spelling, grammar), the mathematical 
skills, and all others which deal VTith data, repetitive- 
observable processes, and mass— developed behaviour pat 

3 

terns. 



Quoted in: Eengt Jacobsson, School P.e forms in Sweden (duplicated). 

International Working Party on Educational Technology and the 
Learning Process, Geneva: May 1970. 

Smith, op.cit . , p. 4. 

A major contribution has been the classification of educational goal 
(with illustrative test items for evaluating their achievement) by 
Benjamin S. Bloom (editor) . in Taxonomy of Educatio nal Objectives^ 
Handbook I: Cognitive Domain ; Mew York: David McKay Co., 1956. 

The major classifications offered are: ''Knowledge, Comprehension, 

Application, Analysis, Synthesis and Evaluation' » pp. 62-200. 
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(c) they tend to be those xtfhich are either most measurable 
objectively s or most readily observed at any selected 
time of testing „ 

(d) they are usually quite general and not clearly-defined 
when they refer to non- cognitive development; attitudes 
toward home and school 5 loyalty toward community and 
country s honesty 9 strength of character® self “mo tivation 
toward learning® respect for scientific concepts® and 
ability to play a role sufficiently independent for per- 
sonal happiness and sufficiently group-oriented for 
social success. 1 Such goals are frequently to be found 
in ministry documents or in. the rhetoric of parents s 
politicians 3 teachers, and, often, the students themselves. 

A Unesco document which shows similar characteristics 
calls for such main goals, among others, as 'values of 

the society' which are interpreted as; 

"value attached to learning, interaction with others, 
adult/ child relations, sex and role differentiation, 
international understanding". 

It further urges: 

"personality development, participation in a variety of 
activities, health and hygiene." 

1 The following classification of non' cognitive skills is given in 

David Krathwohl, Benjamin Bloom, Bertram Masia, Taxonomy of Educa- 
tional Objectives, Handbook II: Affective Domain , New York; David . 

McKay Co. s 1964: "Receiving (Attending), Responding, Valuing, 

Organization, Characterization by a value or value complex'", pp. 95-175 

2 Unesco , Final Report, Meeting of Experts on Curriculum of General 
Education , Moscow 3 16—23 January 1968, Paris, pp. 7 9. 



O 

ERIC 



21 



IIEP/RP/8 - page 17 



(e) such goals as have been described in (d) above, are 
usually not readily amenable for use as evaluative cri- 
teria, testing devices for system effectiveness (formal 
or informal) are quite unreliable,. 

(f) a number of system goals refer to the conditions of edu- 

cational opportunity rather than to the gains made by 
individual students. Thus* goals are expressed in terms 
of 'full and universal' enrolments* 'open access' to 
further education beyond the primary level: free educa- 

tion to all students to the extent that each individual 
can benefit from his participation* 

(g) in a number of school systems where planning for educational 
development is operative to any extent, the goals of the 
system are often described in terms of manpower to be pro- 
duced. Thus, \*e may find explicit numbers of ' specified 
positions for which trained persons will be produced, or 

we may note the more general intention that moire 'X' will 
be trained for the needs of the economy without exact quan- 
tities being indicated. 

Goals become operational as curricula 

The characteristics which are described above tend to be- opera- 
tive as the result of general agreement that they will be given ade- 
quate funding with implementation as curriculum processes. It has 
been, therefore, the task of educators to translate into functional 
programmes the determined goals of the system. And, as the curricula 
become sequences of learning experiences, they are systematically re- 
created as courses, syllabi, term and quarterly 'units', and eventually, 
as 'lessons’, 'lectures’, 'practical exercises', etc., in daily class 
activities. 

It is in this basic form that goals become most tangible. 

Their definition in explicit (or implicit) form becomes feasible, and 
the relevance of goals to conditions of need and reality may be perceived. 
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The basic-, concretized 'courses' within the curriculum enable 
us to note whether there may be goal irrelevancies in the forms of 
'dead' languages, extinct 'sciences', and other uses of educational 
investment of little consequence in the movement toward national 
development. Certainly, such determination may be sought, and found 
with more or less difficulty. An important aspect of evaluation 
ought to concern itself with the gaps between goals and relevancy, as 
well as between goals and goal-achievement. There is, however, no 
reason to exclude from definitions of relevancy those cultural and 
artistic experiences which contribute to personal growth as well as 
to national development. 

It is within the curriculum also that teachers and school sys- 
tem clients begin to evaluate empirically the achievement at near- 
term of system goals. Teachers practice informal testing and indivi- 
dualized, observation. At the higher levels examinations and practical 
exerci-ses are used to determine if goals are being achieved. Pupils, 
parents, school officials, and employers of school leavers also make 
various evaluative attempts, indicating their satisfactions or dis- 
satisfactions as has been noted earlier. 

Yet, with the start of evaluation processes, no matter how 
crude, within the system itself, we note that goals are in need of 
continued study. The following circumstances are generally prevalent: 

(a) some system goals which are stated explicitly are mea- 
sured more or less objectively. 

Bloom states the case directly: 

"If education is to be open, public, and examinable, the 
specifications for it must explicit , and either the pro 
cess of education or the outcomes of the process must be 
examinable in relation to such specifications. 

1 Benjamin S. Bloom, "Some Theoretical Issues Relating to Educational 
Evaluation", in Ralph W. Tyler (editor) , Educational Evaluation: Hew 

Roles, New Means . 68th Yearbook of the National Society for the Study 
of Education, Chicago: University of Chicago Press, 1969, p. 29. 
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We would amend this by suggesting that both process and 
outcomes must be 'examinable 1 , 

(b) some stated goals of the system are not measured objec- 

tively:, but by teacher observation without standardized 
criteria for judgment, L-' : 

(c) the achievement of some expressed goals is not measured 
at all, but is found to be importantly operative in the 
curriculum. We note as an example, pupils' involvement 
in the ’cultural’ areas such as music, the arts, physical 
education, etc. These activities usually are not mea- 
sured as to system effectiveness, although pupils with 
special aptitudes in these areas are noted and encouraged-; 

(d) • the achievement of system goals is of ten measured . at 

specifically determined points and becomes a factor in 
the elimination of pupils from the system; ■, 

■ (e) system goals are often explicitly stated, but in. reality 
are negated from the start by: insufficient primary or 

secondary education facilities in rural areas- poorly 
trained (and not enough) teachers in most of the primary 
schools, failure to enforce such requirements as compul- 
■ sory attendance and secondary pupil vocational guidance , 
operation of non-functional, non— sustaining literacy pro- 
grammes for all citizens needing them. 

'‘There has always been, and no doubt will < al; ways be , a 
great gap between the rhetoric and the deed .. . This does 
not say that the announced aims -were a contrived decep- 
tion." Rather it is a sign of how long it takes, to alter 
the course of an .ongoing educational: system, tO;.r e d6pl°y 
its energies in new directions and to marshal new energies 
and resources to take on new tasks and to do old ones better. 

, . / t . 

^ Philip H. Coombs, The World Educational Crisis: A System s Analysis, 

New York: Oxford University Press, 1968, p. 100. 

O 
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Simply stated, under certain circumstances system goals 
are enumerated but not acted upon. there is, therefore, 
no educational output (or achievement) to reflect such 
goals. 

The relevance of curricula 

Goals cannot be realistic nor can system output in terms of 
student achievement reflect the accomplishment of goals unless the 
curriculum makes such conditions possible. We consider that curricu- 
lar processes, the nature of curricular materials and their use, the 
depth and breadth of student experiences, and the procedures of teach- 
ing all have their role in achieving desired educational outcomes. 

Without going too deeply into detail (since this is not a 
curricular study), we point to a few examples related to curricula 
and goals which may determine effectiveness. 

(a) When the goal is generalizable learning of cognitive 
skills, such learning is not enhanced by rote, memoriza- 
tion, Or repetitive exercises of non— meaningful operations • 

(b) When the goal is to awaken love for literature, traditional 
culture, and the arts, such accomplishment is not encour- 

• " aged by processes in V7hich expression is frustrated, where 
emotional involvement is suppressed, where the arts are 
presented without imagination, and where cultural, liter- 
ary and artistic participation are absent or are under- 
taken in a primitive or coercive context. 

(e) When the goal is better understanding of natural laws, 

scientific concepts, and ecological balance, achievement 
is hindered by strict- adherence to textbooks, lack of 
student involvement with their own immediate environment, 
and lack of a generalized application to school life of 
the concepts of logic, reason, and the scisntific method, 
(d) When the goals are active citizenship, pupil responsibility 
and the development of ethical behaviour, these goals tend 
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to be frustrated when it is not recognized that citizen- 
ship experiences begin within the government of classes 
and schools, that communities in which students and their 
families live are dynamic textbooks for the study of how 
people live together for their mutual benefit, and that 
individuals are the units that make up all systems of 
government. We cannot expect that coercive, dictatorial 
classrooms will prodv e citizens who are competent to 
participate in developing democracies. 

While the above discussion is focused upon the curriculum and 
educational methodologies, the nature of school systems output is de- 
termined by such processes. A major problem is to ascertain the 
quality (and quantity) of such goal achievement as has been cited here* 
When we review the two aspects of goals, the cognitive and the 
non— cognit ive , we can discuss the means used to evaluate student 
achievement in each of the areas. 

Evaluation of goal achievement: cognitive skills 

First, as to cognitive goals, the general approach has been 
through testing procedures which begin with the periodic observation 
by teachers of the nature of each pupil’s ability to satisfy certain 
individually-determined requirements . These include ? reporting back 
to the teacher what the teacher stated at some earlier time- report- 
ing to the teacher whas was found in school textbooks: carryings out 

assignments (homework) as related to the textbook or the presentatibn 
by the teacher* 

Iii determining pupil accomplishment of cognitive skills teacher- 
developed, or school-developed, tests are usually a first phase, taking 
into account the specific experiences of pupils in the particular 
classes. 

lluch of such testing deals with short— term experience, basically 
covering several days or weeks of schooling. In many ; j?ays such testing 
provides school officials (including teachers) with current data about 
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their pupils, about teachers' relative effectiveness, and about the 
educational needs which exist in the localized situation. As may be 
determined by perceptive educators, methodologies used and curricu- 
lum processes may receive careful study, remedial courses may be in- 
stituted, and better use may be made of existing materials. Thelen 
describes the latter use of evaluation in referring to ’diagnosis 
and trouble-shooting’ activities of evaluators and teachers. 

"Trouble-shooting is built-in feedback that enables the 

teacher and the group to see how they need to modify their 

. . : -l 

activity ... 

On a. somewhat higher level, eliminating the dangers' of bias' 
for or against a given school, class or pupil, are the examinations 
prepared by external examiners on the assumption that there is a stan- 
dard body of knowledge to be taught and learned, regardless of loca- 
tion of school or teacher qualification. 

This testing may also yield feedback information which will 
then encourage educational change. It may, however force teachers 
into establishing ever-more uniform curricula and teaching methods. 

We see the potential for a paradoxical situation, therefore: for 

schools of poor quality and inadequate resources, such examinations 
may point the way to up-grading of teachers and curricula. For 
schools of relatively competent teachers and: psychologically-sound 
(if unorthodox) methods, such testing may serve to enforce conformity 
to outmoded or undesirable procedures. 

To review further the procedures for evaluating achievement of 
cognitive goals, we find that testing encompasses not just individual 
schools, but entire systems. There are generally, again, two aspects 
of such testing. First, is the ’summative’ which is prepared year 



Herbert A. Thelen, "The Evaluation of Group Instruction”, in 
Ralph Tyler (editor) op . cit . , p. 148. Also pp. 124—127 hhd 
pp. 151-155. 
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after year by specialized, educators to foe used at the completion of 
specified blocks of school time- This may take the form of examina- 
tions like the English ’Eleven-plus T and General Certificate of 
Education, the French Baccalaureaf, the New York State Regents ex- 
aminations or other certif icate-earning , schbbl-leaving tests. 

VJhiie such testing procedures are often seen as gauging the 
effectiveness of school systems, they actually do no more than assess 
pupil status at a given point in time. ' There is considerable evidence, 
in fact, that j such testing cannot reliably evaluate system effective- 
ness since ^hipil achievement is determined by their experiences in 
school s their experiences out of school, as well as by numerous social 
factors, economic aspects, and motivational status, all generally un 
related to school functions. The work 6f C61eman s Huseh, and others 
tends to bring these aspects into prominence. 

The second aspect of systems— wide testing is that commonly 
referred to as ’standardized 7 . Simply stated V through successive 
application of tests (and sub— tests) it is determined that students 
of average competence in a specific skill will receive ah average 
score on the resultant test. Thus, a standardized test measures 
achievement as it might be demonstrated by normal population of stu- 
dents, assuming a common (not necessarily uniform) hurtituium arid 
competent teachers. Standardized tests have been used throughout the 
United States, and with modifications in other western hemisphere 
countries. The results of nuch ’formcitive 1 tests 'have been useii both 
to determine the achievement of individual students (and their reme- 
dial needs) and the presumed effectiveness of school systems. 

Such * effectiveness T , while evident in terms of pupil status, 
cannot be realistically used in terms of the system itself when the 
existence of intervening factors which inhibit or restrain pupil pro- 
gress Is acknowledged. 

In the various types of testing procedures described above, 
we note that: 
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(a) they are basically related to individual pupil status, 
though as aggregates they are used to define system 
effectiveness s 

(b) they are indicative of specific curriculum content, 
generally rote-learned, and non-generalizable, 

(c) they are often used as guidelines to support policies of 
pupil repetition or elimination from the school system, 

(d) they are usually quantifiable so that scores may be 
assigned, though the closer to the teacher and the school 
the testing is, the less objective may be the quantifica- 
tion; 

(e) they do not take into account the influences upon pupil 
achievement of the ’informal' educational system which 
makes it difficult if not impossible to evaluate with 
such procedures the effectiveness of the formal system. 

Test ing for incremental gains? the value-added concept 

It seems to us that the common approaches to measurement of 
student achievement of cognitive skills have two basic values? first, 
they indicate the status .of groups of students at determined points, 
and secondly, they make it possible to attempt some form of assistance 
to individual students. We do not believe, however, that such terminal 
or cross-sectional measures can reasonably be applied to the effective 
ness of school systems, or even of sub-units within the systems. There 
are too many intervening variables. 

However, we find numerous instances, particularly in the United 
States, of the adaptation and use of achievement measures on a pre- 
test, post-test basis. Appropriate tests are administered to a given 
student population at an early point in the programme to establish a 
base-line, or first level of status indication. Following a period 
of instruction, a comparable test is administered andthe differences 
of achievement are noted. „ Flanagan describes the use of such procedures 
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for testing student progress ^ith newly-developed instructional me- 
thods and materials. 

It appears probable that the gains (or losses) in achievement 
as demonstrated by aggregates of students would be reliable indicators 
of the effectiveness of their instruction. Bloom has pointed out; 

... unless the criteria of ■ effectiveness are related to 

changes (our italics) in students, the researcher has avoided 

2 

the primary criterion * .. 

This is not to say that social and economic variables; may not 
suppress or enhance s t udent ach ieveinent apar t from the . e£f ect s of. 
schooling* Such variables can, however, be held constant and allowed 
for with large mxmbers of students in coaanon circumstances , their ex— 
posure: to common instructional and administrative processes , and the 
pre-determinea short-term between pre-testing and past-testing. - 

With growing demand for school account ability,.; the calculation 
of educational v*, !> ue— added by such procedures can give .indications of 
the results of 1 schooling. Such calculations can be more readily ana- - 
lysed within known! contexts of pupils 1 soc itrr economic .A evels : or prior 
school experience* Given the establishment: of- a ibase^lirie achievement 
level and the application of education processes?; the fx>irx*7ard mpyer 
ment by individuals and groups (or lack of movement) 1 may be. seen as 
a function of the school and the system. Thus, with certain cognitive 



John Flanagan , "The rises of Educational Evaluation in the 
Development of Programs, Courses , Instructional Materials, and 
Equipment , instructional and Learning Procedures , Arid Administra- 
tive Arrangements", in Ralph W. Tyler (editor) , op.cit . s ! pp. 225—228. 
Benjamin Bloom, "Testing Cognitive Ability and Achievement M , in 
N.L. Gage, (editor); Handbook of Research on Teaching , Rand, 

McNally and Co. , Chicago , 1963 , p. 379. 
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skills , it is feasible to attribute pupil progress to system effective- 
ness. It would be difficult to ascribe effectiveness with similar 
confidence to the results of such end-of-sequence testing as ’Eleven- 
plus', Baccalaureat , or other procedures which do not permit a value- 
added determination. 

Evaluation of goal achievement; non-cognit ive skills 

As we have noted earlier, it has generally been in the area of 
specific, cognitive skills (knowledge, problem-solving, higher mental 
processes) that efforts in evaluation are quantifiable and objective. 
From such testing the concepts of student success and of system effec- 
tiveness have been developed. 

Mention has been made above of the areas of educational goals 
which are non-cognitive, i.e., referring to attitudes, values, apprecia- 
tions, interests, and personality characteristics. These, too, are 
expressed as goals for educational effort. 

Although there is even greater controversy over the formation 
and development of such behaviour components in school environments . 
school personnel have generally tended to accept this task within 
their roles. Psychologists, sociologists and others interested in 
the development of human attributes have generally indicated that 
these traits are socially formed, starting with the home and contin- 
uing within the culture of the family and community. Further, they 
contend that even within the school, there exist peer-group culture* 
and teacher orientations which are non-cur ricular and which affect 
the development and expressions of non-cognitive patterns of behaviour. 

This question is considered from another standpoint by 
Krathwohl, Bloom and Masias 

"While the psychologist and philosopher may have views on what 
is desirable and even necessary in the affective domain, there 
is still the question of what affective objectives society 
will permit and even encourage. Our own society has fluctuated 
as to the affective objectives it will permit the school to 
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develop. Political and social forces arc constantly at work, 
pressing the schools for some affective objectives and just 
as constantly placing restrictions on the school with regard 
to others. • 

Yet, since the school system is orientated toward the develop- 
ment of both intellectual and nonrcognitive skills, several problems 
of process and evaluation should be noted. 

In the acceptance of educational goals svich as improved atti- 
tudes, loyalty, character, appreciations, etc.,, there have been few 
instances of curricula which are sr cifically designed to .’ teach' 
such skills. In some countries, courses in civics, national h.i, story , 
and national culture attempt to deal with or inculcate the desired 

behaviours and practices through specific activities. In general , 

however, the development of such skills is dealt with in the context, 
of the existing disciplines (the cognitive areas) and in terms of the . 

expressed behaviour of the students- ; J .. . . .j 

The evaluation of student achievement in the non-cognitiye 
areas is generally through the awareness and recognition of teachers 
and other competent observers. Attention i.S paid, to the behaviours . , 
of students in their classes and in school-wide activities, as well 

as in such specif ic aspects ;as participation and creativity* . . .. ... 

Teachers and other obsenrj^.;£4Eta. students’ demon- 
strated interests in learning, tbeir i^tivatipn to study and go forward 
in the direction indicated by; the. teacher, apd, their persistence in 
applying themselves to required procedures-.. -Teachers may designate 
such behaviour as ’good character 1 , ’favourable attitude toward school’ , 
or ’ self-actualization’ within a specific environment . These observa- 
tions are clearly subjective, .-generally;.. expressed in terms of teacher , 
expectations and needs, and without reference to school processes. 

It does seem feasible, however , to assume that students who function 
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in these ways within an educational system are highly compatible with 
the system as to expected behaviours and indications of success. 

They and the system may be seen as ‘effective' . 

Since such evaluation is difficult to quantify and generalize, 
we can describe student status in this way, but further study must be 
done to ascertain the processes within the system which may have pro- 
duced that status. Erathwohl , Bloom and Masia suggest that: 

m ... if specific changes are to take place in the learners, 

. . . the learning experiences must be of a two-way nature in 
which both students and teacher are involved in an interactive 
manner, rather than having one present something to be learned 
by the other. 

Xfe may need to look, with this regard also, for the effects of non- 
system variables before we can claim system success for student 
adaptability. 

Student participation as an indication of non-cogn itive 
development 

Student participation may be seen as an indicator of system 
effectiveness under specific circumstances. The level of student 
responsiveness within .classes, student feedback tp; teachers, and the 
openness of student participation in discussion all may indicate 
positive attitudes , character development, and so forth, provided 
that such student involvement and expression are given opportunity 
within a school system. Student achievement in the non-cognitive 
skills can be observed in 'open' systems where teacher attitudes are 
receptive and encouraging. In such circumstances procedures can be 
developed for quantifying the nature of student participation as has 
been described. Observational techniques coupled with rating scales, 
case studies, and sccio-metric methods may be applied to determine 
student status, and, inf erentially , to establish school, and system 




1 Ibid . , pp . 81-82 . 
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status in this area. With the further inclusion of pre-test, post- 
test concepts, such procedures may be relatively effective in noting 
school-determined gains. 

Xt is* of course , conceivable that student participation in 
class inter-action may be usefully carried forward to collaboration 
in school operation, and ultimately to system evaluation, modifica- 
tion and reform. 

Student creativity as an indicator of non-cognitive de velopment 

Student creativity, while initially requiring the achievement 
of skills in manipulation of specific communications tools, may be 
seen, when evident , as an indicator of non- cognitive achievement. 

Thus, when we recognize that the basic approach to literary expression 
is in the mastering of the cognitive skills of reading, writing, spell- 
ing, we find that the opportunity for and encouragement of such ex- 
pression nay indicate system achievement. Similarly, the involvement 
and exposure of students to graphic and musical arts within curricu- 
lar experiences are based upon system goals and require evaluation as 
to effectiveness. Such effectiveness may be determined by careful 
study of the conditions and nature of student expression in the arts. 

We recognize that within given systems there may be no curri- 
culum for ’literary* creativity, or for graphic or musical expression 
on a creative level. When students produce such works , therefore, we 
are almost required to search in the school processes for incentives, 
motivations, and positive attitudes in these areas. Ultimately, we 
may determine that this is such an individualized matter that the 
educational system, or any of its elements .plays no role in. this 
development* 

On the other hand, when we note full courses, specialized 
teachers, and available materials, we may look for evidence of both 
appreciation and specific creative expression in some form of output. 
As to appreciation, some common (if depressing) practices include 
formalized testing for recognition and memorization of works, artists. 
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themes, and so on. Further evaluation may cake note of self-develop- 
ment in leisure time, selection of undirected reading choices., and 
post-school interests and activities. 

Indicators for system evaluation 

In the developed countries of the wo.rld, evaluation of school 
systems is carried out on two bases; first, on the qualifications 
of school leavers in terms of years completed, evident literacy, and 
certificates and degrees held; secondly, on the internal system evi- 
dence of achievement scores coupled with repetition rates and drop 
outs. 

Little use is made thus far of the less quantifiable forms of 
system evaluation when they are voiced by school clientele, including 
students. The concepts of accountability are still rather limited to 
personalized or empirical determination except in rare cases. These 
occur following discussions of 'out-moded' curricula, lack of concern 
for individuals, and excessive stress on competition, elimination, or 

discriminatory tracking (streaming) . 

An interesting concept of accountability has developed in the 
United States, as reported in a recent issue of the Fhi Delta Kappan. 
Private companies are given contracts to operate educational programmes 
within schools. Such ‘performance contracts' provide a sliding scale 
of payments in accordance with the rise in pupil achievement levels. 

The programmes are supported with national funds to the school dis- 
tricts which then arrange for the contracts. Companies are responsible 
for providing teaching personnel , teaching materials, and for deter- 
mining the instructional methods, Thus accountability becomes a matter 
of profit or loss to the contractors. They are indeed accountable , , , , 

since non-achievement by pupils to contracted-f or levels may result 
in no payment. Other accountability projects will: range from trading 
stamps (exchangable for gifts) as regards to students to cash bonuses ;il 
for teachers. - : : 
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A Canadian experience with accountability dates, back to the 
late 19th century when financial grants to high schools in Ontario 
were made dependent on the number of pupils who passed an examination 
after a year or two of attendance. Phillips reports that. 

’’under full pressure of payment by results, teachers and pupils 
began rapidly to measure up to the requirements ...One reason 
for this amazing improvement in efficiency was that trustees 
were aroused from lethargy. Men their school failed to earn 
grants, their response was to blame the senior master and to 
dismiss him . . . 

"No doubt some of the teachers dismissed were lazy or inefficient. 



But nearly all were shrewd enough to take advantage of every 
new means that was offered to get results. Teachers' profes- 
sional journals were filled with sample examination papers, 
model answers, and advertisements of little books containing 
notes on various subjects- the memorization of which would en 
sure success on the examinations. History teaching became the 
application of a system of mnemonics and the teaching of litera- 
ture little more. 



"Experience with payment by results in Ontario proved that it 
is possible to raise standards quickly if the criterion is de- 
fined as mastery of prescribed content. But there was a storm 
of protest against the sacrifice of all other educational values 
for the attainment of this end. T.n 1883 payment by results was 
abandoned in the province • t5 

In developed countries there are serious moves toward the re- 
thinking of system goals. This is evident, as well, in the United 
States, Germany, and the United Kingdom where the varied centraliza- 
tion would not seem to encourage national, or system-wide agreement 
on go a: Is . 
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The discussions have generally led to consideration of curri- 
cular processes and, equally important, to attempts at determining 
indicators of the effectiveness of educational systems. 

In developed countries such indicators have been designated 
as: (a) relating to system accessibility; (b) relating to system 

processes (c) relating to the educational product. 

(a) As regards system accessibility , we would consider as 

valid the following indicators: 

(i) a truly open and available primary system at no cost and 
with appropriate subventions to those families for whom 
opportunity costs were to be determined as excessive: 

(ii) a clear-cut policy on promotion so that failure to achieve 
according to determined standards would not result in re- 
petition, but would bring about compensatory (and remedial) 
services and modification of curriculum, 

(iii) the development of educational and social procedures which 
would reduce the disparity at the start between poor chil- 
dren and middle class or others. Such procedures could 
provide earlier educaticnal experience, increase family 
literacy and motivation, relieve mothers of excessive 
child-care burdens by providing ancillary nursery or other 
services . 

(iv) the establishment of close relationships between primary 

education and the next level through information, guidance 
and orientation of pupils and their families. This would 
enable families to have specific understandings of the 
possibilities for their children in continuing their edu- 
cation. It would enable personnel in the primary schools 
to convey information to their students and to encourage 
appropriate further development. It would enable secon- 
dary school personnel to know and understand the condi- 
t ions of primary education, to appreciate the status of 
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incoming pupils » and to restructure their own curricular 
and guidance procedures, 

(v) the determination of a socially appropriate and economi- 
cally feasible compulsory school age so that students 
who wish to do so may leave ; the school. This could only 
he established together with provisions to cpunsel stu- 
dents before they drop out, for incentives and reasonable 
conditions for remaining in school, to serve them with 
employment advice and liaison,, arid provisions for further 
education when desired. 

(b). As regards system, processes , indicators of effectiveness 

would include* . : 

(i) content in curricula, courses and programmes which is 
developed in terms of agreed-upon goals reflecting 

, national and individual needs . This indicator includes 
relevance for students and their families, acquisition 
of skills which are in keeping with concepts of self- 
development as x?ell as with the manpower market, incor- 
poration of materials which are reflective cjf national 

as well as community interests as, regards tradition, 
culture, and, apt ions for future development. With curri- 
culum content as an indicator of , ef f ec tivene s s , wp . can 
extend our, study to relationships between system organi- 
sational echelons and the. actual implementation of sys- 
tem; goals*;), ,v.. .. • 

(ii) teaching, practices; in the. curriculum which are oriented 
toward student status and . conditions of learning. It is 
unrealistic to consider curriculum content, as an indica- 
tor of effectiveness without referring to methodology as 
well. The translation of educational goals into content 
can only be made concrete through student experience. 
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Methodology involves teaching practices, uses of materials, 
as well as general aspects of relationships between the 
teacher and the taught . 

We would stress that the development of teaching prac- 
tices to convey curricular content would need to make 
full use of relevant psychological factors. The acquired 
body of knowledge we now possess about how learning takes 
place must serve as a foundation for methodological de- 
velopment. Thus the improved procedures for teaching 
and learning may serve as indicators of system effective- 
ness when related to goals, curriculum content and to 
measured output . 

(c) As concerns the evaluation of t he educational jn^oduut as 
indicators of system effectiveness, we would look to the following: 

(i) establishment of procedures to reduce to a minimum the 
number of school leavers prior to the basic certificate 
stage as may be determined by national policy.; 

(ii) the utilization of interventions for measurement of edu- 
cational value added within the school system at numer- 
ous points. The 'product' therefore would be measurable 
in terms of gains over initial, base-line status; 

(iii) the development of quality criteria for the evaluation 

of manpower output. The production of engineers, machin- 
ists, nurses, or personnel of any profession is now seen 
in numerical terms, i^e . , how many were produced. An 
important indicator of effectiveness is the qiality of 
the production, not only based upon courses taken, years 
spent, theses written, or practical activities pursued, 
but the varying levels of efficiency of the certificate 
or degree holders at the end of their preparation. As 
we have noted earlier, the production of qualified per- 
sonnel involves a minimal passing group with numerous 

O 
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individuals on a quality continuum from ‘passing* to 
'outstanding'. The evaluation of . such differences would 
proye a meaningful and .useful indicator of system effec 
tiveness, allowing for returns to the curricula for modi- 
fication or reform;, : . • • 

(iv) the organization and operation of research and develop- 
ment units which would undertake needed studies, provide 
recommendations, and participate in implementation. A 
typical task . of an R and D unit would be the preparation 
of long and short term follow-up studies of school leavers, 
including graduates. Such studies should attempt to. de— 

. , termine the effects of schooling upon Personal mobility 

ai*d professional effectiveness. 

Naturally, we assume that the longer the term of study, 
the more impressive would he the effects of non-school 
variables, including chance , war , revolution, and econo- 
mic movement s . Yet , with such studies, and the develop- 
ment of controls, we may determine the critical fac ors .. 
of school experiences which, have determining influence 
in spite of outside variables. 

Summary • \ 

In the foregoing discussion, we have attempted to establish ^ 
directions which might be taken to evaluate school system effective- 
ness. -We have not mentioned the elements of cost benefits or cost 
effectiveness since such considerations ^are limited to economic and 
financial aspects. We are interested in. the possibilities of estab- 
lishing criteria for evaluating syst^ effectiveness which may, when 
coupled with such economic studies, be useful to educational planners 
throughout the system. We include with them those who help establish 
goals as well as those who plan and organize administration of schools, 
school curricula, teacher education programmes, and le, arn f n 8 ? r °?® dur ® s * 
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We nave stressed in this discussion the potential for the 
evaluation of school effectiveness indicators in developed countries 
primarily because of existing organizational strengths and available 
financial resources. In our view the developed countries are gener- 
ally nearer to the points of universal education and qualified teach- 
ing staff so as to begin to implement the concepts expressed. Finally 
we note that the research and development capabilities in developed 
countries make possible feasible involvement in such activities. 

They possess personnel and research facilities which can be engaged 
in the types of studies, programme development, and heuristic pro- 
cesses which are required. 

For developing countries j we suggest that such considerations 
may also be valid and realistic. Given the existing conditions of 
insufficient resources, low enrolment rates , high wastage rates , 
poorly qualified personnel, and inappropriate curricula and metho- 
dologies, it is desirable that study be usefully applied in specific 
areas. Technical assistance teams for research and development can 
offer studies to establish relevant priorities. Questions of goal- 
setting should be resolved so that ensuing development of curricula 
and personnel training can be undertaken within the recognized con- 

straints. 

When efforts were made by developed countries, too often such 
practices and techniques were lifted almost in toto and set up for 
operation in developing countries. This is: now almost universally 
recognized as ineffectual, if not counter-productive. We look in- 
stead to the establishment of improved procedures in educational 
systems which will serve as 're-design* models for developing coun- 
tries. Such models would be oriented toward the generalizable con- 
ditions of developing countries and would enable the reconstruction 
of system elements if not the creation of new ones.,. Thus, we might 
find reconstruction of planning departments, of teacher training 
sub-systems, of primary education curriculum and methods, etc., all 

O 
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amenable to modification as parts, yet within a total system confi- 
guration. The kinds of changes we have conceived of in developed 
countries could serve as an aid to developing ones so that repeti- 
tion of errors or malfunctions could be avoided and some 'giant-steps 
could be taken • 




42 



PART II 



CURRENT PROCEDURES FOR EDUCATIONAL EVALUATION 



o 

ERIC 



IIEP/RP/8 - page 39 



Introduction 

In the preceding discussion we presented a synthesis of the 
activities and problems concerning the evaluation of educational sys- 
tem outputs. We described the nature of current = considerations and 
the developing pressures which are affecting the thoughts of resear- 
chers in the field* 

It is important to note t-'-iat both economists and educators 
are seeking to determine the differences between the evaluation ;of 
students 1 achievement and the evaluation of systems- as they enable 
students to achieve. We note also that the bulk of past research in 
educational measurement has been primarily focused on the achievement 
of students in cognitive areas. As economists, and increasingly, the 
general public, have demanded systems evaluation, educators, politi- 
cians, and systems managers are beginning to seek a more common under- 
standing and basic definition of the problem. 

Existing cohditions of practice in the evaluation of 
school system outputs 

In this section of our study we propose to review the evidence 
of existing practice in the evaluation of educational outputs. We v . 

shall describe the following categories: i 

'• •• -J the purposes of testing’ in educational systems, 

- the types of evaluation instruments -used; v« 

- the frequency of testing and uses made of the data obtained, 

- implications of the foregoing. 

The purposes of testing in educati onal systems 

The following conclusions Wdy be drawn as to the purposes 

(explicit a!ttd implicit) "Of achievement testing: 

: ' v -- ‘ (d) 5 "to relate pupil learning to teachers’ expectations, 

(b) to determine the ability of students to report and apply 
? 3V inf6rmatibii which has been learned and to demonstrate 
learned attitudesand desired behaviours 5 on 
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(c) to establish evidence for the elimination of students 
from the school system, or for the repetition of years, 
or for assignment to special tracks or schools > 

(d) to support the admission (or refusal of admission) of 
students to further levels of education-, 

(e) to serve as a basis for the award of certificates, degrees, 

or licences , 

(f; to provide students xtfith an additional motivation as well 
as further learning opportunity by means of a test 

situation, . 

(g) rl to establish individual student status within a group, 
'group status in a school, etc., as regards a prescribed 
curriculum. Such status may be reported as class posi- 
tion, test score, percentile of the tested group,, or grade 
placement in a standardized norm* 

The foregoing purposes for achievement testing of students are 
common to all countries, developing as well as developed. They should 
be read, however, as a descending order of frequency. Thus the first 
five items (a-e) are generally found much more frequently, whether 
expressed or implied, than the later purposes. 

In addition, we find a set of purposes for achievement testing 

which are quite inf requent , being notably operative in .some British 

and American schools: . .. . 

(a) to determine whether the teaching (and the teaching staff) 

has been effective;- • v, . ■■= 

(b) to establish within a school -community relationship the 
concept of accountability* i.e. 9 that the .system is j^espon 
sible for pupil learning and will see to it that evidence 
of non- 1 earning is studied and used, ior change j. 

(c) rf to pei*tni": the modification of curriculuin .or.nioves toward 

new curriculum development in terms of evidence that poor 
ces:" results have come about within an existing curriculum 
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(d) to permit the re-thinking of teaching methodologies so 
that following poor test results* the methods which were 
used with low-achieving students might be improved, 

(e) to establish a relationship between the results of 
achievement testing and the effectiveness of school 
systems • 

As we noted earlier, the foregoing uses of achievement test- 
ing are less frequent than the earlier list (supra) , and these, too, 
are in a descending order of frequency. In fact, the final item re- 
fer- ing to system effectiveness is not currently found through re- 
search or systematic study. It is, rather, occasionally used as a 
critical concept: that the schools are doing a very poor job alto- 

gether, look at the poor achievement of pupils’. In a number of 
countries this evaluation of local school systems finds expression 
in migration from rural to urban areas, and by the movement of middle- 
class families to suburban districts and private education. 

The types of evaluation instruments used 

All evaluation of pupil achievement begins with observation 
by teachers and parents, though on occasion employers and other 

interested persons offer evaluation as well. 

We find, therefore, the following measures of evaluation most 

generally used: 

(a) teacher observation: pupil achievement is reported as 

observed or in retrospect, summatively, 

(b) teacher observation: pupil achievement is reported with 

specific forms, rating sheets, or other standardized 
materials , 

(c) in-class tests of current achievement, provided as part 
of textbook or workbook materials, 

(d) in-clas? tests, prepared by teachers periodically as de- 
termined by the content of the curriculum and the lessons 
(pupil experiences) which had been completed. 
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(e) external examinations, prepared by individuals or committees 
with special expertise to evaluate student learning, ability 
to express knox*ledge, and ability to conform to the exigen- 
cies of testing conditions 

(f) external examinations, standardized x^ith relation to the 
requirements of the curriculum and to the quality of the 
student population of a common age, experience, and ability 
in the subject matter of the examination. 

In the preceding list, we find no specific order of incidence 
of objective or subjective measures. The teacher who reports obser- 
vations ad hoc is more subjective, hence less reliable, than the 
teachers who are required to report their observations qf pupils on ^ 
a form, a rating scale, c three-point choice, or other such ‘gliding’ 

device. 

On the same basi J , In - class tests of pupil achievement may be 
subjective when they are: (a) teacher-made; <b) responded to by 

pupils writing freely; and (c) when they are scored by the teacher 
who prepared them, stress the relative objectivity (that is, 

there is no questior. >c to what the student ’ s. answer is) of tests 
which call for decisions on one of three possible answers, a 'true/ 
false’ choice, a matching of related items, or a completion item in 
which a single word or phrase is to be inserted. Mot withstanding 
the rigidity of such testing instruments and their indication of 
limited -spects of learning, they do permit non-biased evaluation 
and generalizations about groups of students. Similar comments may 
be made with regard to tests which are : included in textbooks. In 
the latter case, however, the testing- and-learning potential is also 
evident since the book is available to the student, for checking, re- 
view, and further study. 

In general, there is an attitude prevalent among researchers 
which favours the adherence to objective procedures so that those 
aspects of a stud .c which are not being tested will not intrude. 
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We refer to student personality, verbal fluency, behaviour under stress, 
relations with the t feather or with examiners, and so forth. On the 
other hand, the ability to satisfy an oral examination jury, or to 
present an extended essay or dissertation with no prior knowledge of 
the topic to be selected is seen by many as a roost relevant evalua 
tion of student competence and of system effectiveness as well. The 
problem is not easily resolved except as may be determined by condi- 
tions o£ explicit system goals, of adequate resources in personnel 
for teaching and for examining, and of system openness. Also to be 
taken into account are the effects of the ’informal' educational 
system and of the non-school variables we have mentioned earlier. 
Frequency of testing and uses made of the 
data obtained 

Traditionally, testing of student achievement has taken several 
very specific forms: (i) testing of students as they completed a 

school form So as to permit their advancement to the next; (iO test- 
ing at the conclusion of a school period or cycle to indicate comple- 
tion of requirements for a certificate and for entry to a continuing, 
higher cycle; (iii) testing at the beginning of a new educational : ... 
period so as to confirm entry to higher levels of study. 

Thus, pupils, parents, and teachers have known,, :ia all coun- 
tries , the official and regional examination times. As At iy eh notes: 

■’ '"(Quantitatively , in many countries, a good student during a 
career of 12 years normally site for approximately 16 formal 

examinations , each covering a variety of topics w" w 
Attached to, and usually in conjunction with testing, has 
been the continuous assignment of marks by teachers to students, 
presumably on the basis of objective evidence of accomplishment and 

behaviour. 



1 Naim N. Atiyeh, "Examinations: Trends and Prospects", in Joseph 

A. Lauwerys, and bavid G. Scanlon, Examinations. The W orld Year- 
book of Education, London: Evans Brothers Limited, 1969, p. 376 
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There is a developing tendency to reduce the frequency of re- 
quired examinations, in fact, to eliminate them altogether. In 
Swedish schools 1 , continuous assessment by teachers is now the norm 
with no compulsory xnritten tests or final examinations in the nine- 
year comprehensive school. Standardised tests are available at the 
teacher's option and it is interesting to observe that the use of 
these tests has increased to the point where 90-95 per cent of all 
pupils in the comprehensive schools are taking tests. In the gymnasium , 
which follows the comprehensive school, tests are compulsory for stated 

forms and are spread out from December to May. 

In the Federal Republic of Germany a similar condition is de- 
veloping, with emphasis on teachers' marks, a minimum nambeir,.of re- 
quired internal examinations, and, up to the completion of secondary 
education, no compulsory tests. Research has shown that previous re- 
quired tests for entry to secondary education were no more valid than 
the teachers’ continuous assessment. 2 It is only at the completion 
of secondary education ( gymnasium) that the Abitur examination is 
required. Standardized tests are relatively not used except for 
primary school admission as devices for prognosis. 

The frequency and nature of testing in France is reported by 
Legrand, who notes the conditions which persist despite reforms: 

• The reforming zeal, devoted first to the educational 

structure and now to the definition of the syllabus and teach 
itig methods, has not yet touched the examination system. So 

it continues as before, rendered even more complicated by the 

3 

reforms , . . n . 
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1 Sven-Eric Henricson, "Continuous Evaluation in Swedish School**’-, 
(duplicated). National Board of Education, Stockholm, 30.IX*196$ 

8 p. 

2 Reported by Karlheinz lngenkamp in tahwerys and Scanlon, op.clt . 
pp . 140—145. 

3 Louis Legrand , in Laiiwerys and Scanlon, oj>.cit . > pp. 123-129. 
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Thus the French child goes from test to test - some optional 

but mostly taken as if compulsory: 

Examin d* entree en 6eme : for entry to the f’tat level of 

secondary education, at about ages 10“ 11 ; 

Certificat d' etudes primaires : at completion of the primary 

period, at about age 14., 

Brevet d' enseignement du premier cycle : at the end of the 

first level of secondary education, at about age 16; 

Brevet d' enseignement prof essjonnel : at the end of the secon- 

dary vocational (short) course, at about age 19 > 

Then at the end of the secondary vocational (long) course, at 
about age 2Q, the following choices: 

Baccalaureat des techniciens s 

Brevet. Indus triel or commercial ; {: :.m 

Certificat d f aptitude prof essionnelle^ 

Finally, at the end of the academic second secondary cycle, 
at age 19, the Baccalaureat . 

In reviewing, this situation, Capelle proposes that a single 
certificate be awarded following the completion of compulsory studies 
and that it be on the basis of observation and guidance. He sees the 
value of an examination at this poi^t only as an arbitrating instru- 
ment where parents disagree with the proposals of a school orienta- 
tion committee. - • ... v :■ i <X i v ; s ■ : 

Gapelle: approves examinations for vocational qualifications 
since they constitute "attestations of capability recognized by con- 
tracts of employment ... Thus, a "conventional examination seems 

indispensible • " 

To indicate completion of the ’long’ general secondary course 
he suggests a generally standardized programme of teacher marks which 



Jean Capelle* Tomorrow’s Education. 
Pergamon Press, 1967, pp • 124—144. 
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provide a summary of the schooling completed. For entrance to uni* 
versity level studies, then, a national written examination would be 
compulsory, while university authorities could still select their 
students from within the group completing this test successfully. 

Thus the proposed improvement becomes 0 with a limited number of places, 
a highly competitive matter. 

In England and Hales, the basic examination structure requires 
testing of primary school students for placement within secondary edu~~ 
cation and further testing at the end of secondary education with 
either the General Certificate of Education (G.C.E.) or the Certifi- 
cate of Secondary Education (C.S.E.).* The G.C.E. is a pass/fail 
examination generally taken by candidates for higher and further edu 
cation. This examination is prepared and administered by eight inde- 
pendent boards for students of grammar schools, i .e . , those considered 
to be in the upper 20 per cent Of the intelligence range. 

The C.S.E. is prepared by fourteen regional examining boards 
for the next lower 40 per cent of the intelligence range. Whalley 
describes the three modes of C.S.E. examinations which seem to bring 
it closer to the needs and interests of regional conditions. Be de- 
scribes Mode 1 as an external examination on syllabuses produced by 
the regional boards Mode 2 is an examination based on a school s own 
syllabus 5 Mode 3 is an examination by individual teachers of their 
own candidates using their own syllabuses. All oz course are with 
the supervision and approval of the regional boards on which are re 

presentatives of the national Schools Council for the Curriculum and 

.2 ...... 

Examination. 




Henry G. McIntosh and Daisy M. Penfold, in Lauwerys and Scanlon, 
op . cit . , pp. 110-119. 

G.E. Whalley, The Certificate of Secondary Education , Leeds: The 

University of Leeds, Ins tit it e of Education, March 1969, pp . 16-19, 
41-47. 
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The primary examination, which is still required in some parts 
of England (the 11+) > includes a test of intelligence, a test of pro- 
ficiency in English, and a test of arithmetic. Results, which are 
combined with the cumulative school record, lead to decisions regard- 
ing selection and orientation of students toward various educational 

streams . 

In the United States, teacher-made examinations as well as 
teachers' continuous assessment are found in all states in both pri- 
mary and secondary levels. Tn addition, periodic standardized 
achievement tests are required at various points in both levels, with 
the results’ utilized for student guidance, parent information, and, 
occasionally, as indication of teaching effectiveness. There are no 
s ehoo 1-1 fiav i'ng examinations at the end of primary education, nor are 
there admissions examinations for the secondary schools, which are 
almost all comprehensive. 

For admissioh to universities, examinations are used (prepared 
by the College Entrance Examination Board, American College Testing 
Program, National Merit Scholarship Corporation) in conjunction with 
secondary school records to determine admissions: and scholarship 
awards. More recently, a number of universities have granted admis- 
sion to students with certificates indicating completion of secondary 
studies and strong recommendations from secondary school officials 
or community representatives. This is especially true for members 
of minority ethnic groups without economic means. 

Admission to primary education in the Soviet Union is purely 
by age and no examinations are required. During the primary period, 
continuous assessment by teachers is the rule. End-of-year examina- 
tions are to be found in some schools, but they are optional and are 
locally determined. There is no school-leaving examination for the 
primary period, nor are there secondary school admission examinations 
At the end of the five-year (incomplete) secondary period, an 
examination is required in no more than two or three subjects, given 
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by the teachers of the school. This examination is in both the writ- 
ten and oral forms. Following this secondary period students may 
proceed to special vocational schools X'jhere admissions examinations 
reflecting the schools’ particular curriculum are required. 

Students who continue in the general secondary school cycle 
for two more years are required to pass a ’maturity’ examination 
which results in the award of a certificate of completion. This 
examination is also internal, though it may be administered by tea- 
chers of the school who have not actually taught the specific students. 
The ’maturity’ examination covers in its written form, language, 
literature and mathematics. The other subjects of the curriculum 
are evaluated orally. 

As can be seen, the uniform school curriculum in the Soviet 
Union is a basis for uniform testing, but such tests are constructed, 
under supervision, in the respective schools. 

Students who complete the secondary technical programme must 
complete examinations in both the ’maturity’ aspect and in the area 
of their school’s specialization. 

Admission to universities is determined by examinations which 
are taken in the faculty of the student's intended university 
specialization. 

At present, there is continuing discussion as to the need for 
students to be subjected to dual examinations at the end Ol their 
full secc-:dary education: the one in the spring, the 'maturity' ex- 

amination and in early fall, the university admission test of the 
faculty. The tendency of such discussion is to eliminate the former, 
since secondary school completion can be attested to by school records 
and the university admissions examination can continue to serve its 

screening purpose.- . . . • . ■ 

In Japan, since World War II interest in the development of 
tests has continued to grow. Standardized tests are produced by lo- 
cal publishers and, though not required, are used with pupils in the 
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elementary and secondary schools. The results are recorded on stan- 
dardized school record forms. Although these tests of subject matter 
achievement are used optionally in the schools, they respond to a 
strong motivational situation for students who wish to continue their 
education. We note that the tests are available, with keys and man- 
uals, for sale in the Open market. 

Although Japan has a school system much like that of the 
United States (6-4-4), testing is conducted in conjunction with .screen- 
ing for the upper secondary school . The screening is based, on the 
results of previous scholastic records and achievement tests. Third- 
year pupils in the lower secondary school take a pref ectural achieve- 
ment test which is included in the screening procedure. 

Further testing is required for admission to a university * 
usually with tests proposed by the individual institution. These are 
generally objective tests and have been . criticized for their tendency 
to restrict the patterns of student learning in the preceding educa- 

tional levels f 

We may summarize as follows: 

(a) in most countries continuous assessment is done by 
teachers throughout the primary years: 5 1 

(h) there is a tendency to maintain school*" leaving examxna 

tions where they now exist, but to join their scores with 
the cumulative records 5 ! : • 

(c) rv the frequency of primary level testing;. is , diminishing on r ; 

a close-of-year basis, though a growing trend is shown • 
toward standardized achievement testing at specified 

points ; ' ' - r ' l 

(d) the frequency of star t-of -^secondary tests seems to be 

. maintained where they have existed. Xn this aspect j :.*(.« 
• ptfissure is being felt to admit students to ^secondary n r 

education on the basis of the completed primary cycle?., t cc* • 



O’. Reported by: K. Wakayama, in Lauwerys and Scanlon, op .cit . , pp. 223-226. 
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(e) testing in the secondary school is generally stable 
internal tests seem to be the rule* Some uses of nation 
ally standardized tests are seen with increasing frequency, 

(f) end-of -cycle tests are required at the secondary level. 

These serve as certificates or as credentials for further 
education. In many countries, universities require and 
provide their own orientation-testing procedures: 

(g) in almost all countries, the tendency is to reduce the 

J frequency of national, required tests, except as noted 

above in (f ) ; 

(h) although end-of-cycle and admissions testing has been 
usually written (essay) and oral ^ the over-all tendency 
is toward objective, short— answer test items, with a 
lessening frequency of oral examinations. 

It is clear that the data obtained from evaluation of students 1 
achievement are used to assign them to an educational track, to retain 
them for a year f s repetition or to anticipate their leaving the school 
system. In a number of countries more attention is paid to student 
vocational guidance and to orientation of parents. 

Implications 

The developments reported above are associated to a great ex- 
tent with the slow-moving, but coninuous trend toward egalitarianism 
in the social structure, presumably to be achieved by egalitarianism 

in the schools. Reduction of fixed and required examinations and In- 
creased activities to provide educational experiences which will com- 
pensate for socio-economic and cultural inequalities are factors in 
the expressed desire to reduce focus upon 1 elites 1 . The giving of 
exclusive attention to the development of highly selected and: well 
nurtured (educationally) cadres is not eliminated in any country, but 
is decidedly diminished and thus is more in keeping with the expressed 
goals for national systetas. r : v " 
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Inf erenti ally, at least, evaluation of teaching effectiveness- 
is considered as a by-product of student success or failure in the 
evaluation process. There is no evidence that generalized curricular 
studies cr reforms are instituted following study of evaluation data. 

In some countries, structural reforms such as extension of the com- 
pulsory schooling age or introduction of comprehensive lower schools, 
are in evidence , related in part to evaluation data, but even more 
to the continuing clamour for school opportunity and to the rising 
qualifications called for by the labour market. These latter changes 
are occasionally associated also with revision of over-all school 

system obj ect ives * . ‘ 

In terms of the purposes of this exploratory study, that is, 

to determine the means of evaluation of educational output, we find 
the first and foremost attention being given for student achievement 
at some point within, or at the end ; of , the educational system. 

As to the evaluation of the system itself, this is generally . 
oblique or by .inference. The student is evaluated as a learner. 

His achievement as a learner is seen, in s?me situations, as reflect- 
ing his socio-economic or cultural environment, including parental 
and family conditions. In some aspects, educational investments 
(inputs) are considered to be related to achievement: school budgets, 

teacher qualification, teacher salaries, size of classes, available 
and used educational materials, etc. There is little or no evidence 
of studies of base-line achievement data and of ensuing achievement 
of controlled populations, The value-added concept^ is seldom utilized 

in system evaluation. o ' 

Thus we. find, almost: in polarized contention ,, the preferences 
of economics-oriented observers and those of educational-achieveme- C-- 
oriented partisans. The former express their evaluation of educa- 
tional system outputs in tc,( : r of manpower production, unit costs, 
and ultimately cost-efficiency assumption? • The latter seek to CTal- 
uate educational outputs from the quality basis of student achievement. 
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operational relationships between curricular processes and student 
behaviour, and with the contention that system goals should be made 
explicit so that their achievement may be determined and quantified 
for reliable evaluation. 

It would appear that one group concedes the difficulty in 
quantifying quality elements and prefers to develop evaluative con- 
structs for the already-quantified elements. The other group insists 
that determined quantities are not really evaluative of the system or 
of its processes, and maintains that there is a verifiable relation- 
ship at near-term and at long-term between inputs, processes and 
student quality outputs * 

We would admit to being partisan in the debate since we feel 
that there are few remaining areas open to consideration once the 
techniques of cost-analysis, unit costs and manpower output are uti- 
lized. What remains is the calculation continually of ever-more re- 

fined input-output relationships. 

It is our view that a more productive area for study resides 
in the consideration of system objectives which ought to be explicit 
and therefore more controlling in terms of inputs and processes. ■ 
Student achievement in cognitive areas is now quantifiable and, with 

further research, may be more reliably correlated with inputs and 
curricula. Well-controUed studies on a longitudinal basis may pro- 
vide further interpretations for the evaluation of system effective- 
ness. We look also to the heuristic potential of continuing research 

in assessing non— cognitive achievement goals. 

We see, therefore, not a continuing polarization of hostile 
views, but an amelioration of differences and a more useful synthesis 
based upon continuing research in the various areas of school 
functioning. 

In the ifoll owing section of this report, we will describe 
current and developing research to achieve these purposes.' ; ' ; 
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PART III 



CURRENT RESEARCH ON EVALUATION OF 
EDUCATIONAL OUTPUTS 
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Research presently being c arried out 

In this section of our study ue will review the major efforts 

currently being made in the evaluation of educational outr’t. As 
will be noted, our emphasis is upon the qualitative aspects of school 
systems and those aspects which are related to student achievement of 
system objectives. Much of the work reported here is centred around 
student achievement although there is considerable interest in the 
determination of other indicators of system effectiveness. 

Since other IIEP studies have dealt with economic indicators, 

our discussion will refer to these only slightly if at all. 

On the international scale, the major work has been done by 
Husen and his colleagues in the International Project for the 
Evaluation of Educational Achievement - 1 This work was based upon an 
earlier feasibility study of over 3,000 thirteen-year-old students 
in 12 countries with tests in science, geography, reading comprehen- 

siveness and non-verbal intelligence. 

The Husin study (known as the I.E.A.) evaluated mathematics 
achievement in 12 countries by developing nationally-standardized 
tests as established by committees of specialists in each country. 

Two major groups in the secondary schools were sampled. 

(a) thirteen-year-olds, being the oldest students still in full-time 
secondary education in all 12 systems, and <b) pre-university- grade 
students. In addition to the mathematics tests, students responded 
to items about their attitudes and their backgrounds. Additional 
data were supplied about the individual schools and the national sys- 
tem. A total of 133,000 students in 5,450 schools were tested and 
questionnaires were received from 13,500 teachers and 5,400 principals. 



1 Torsten Husen * op.cit . 

2 A.W. Foshav. Educational AchUvements of Thi rteen-year-olds. j n 

Twelve Countries, Hamburg: Jr.esco, Institute of Education, 1962. 
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Husen describes the operations of the secondary mathematics 
study and reports that : 

" ... it was possible to study the 'effectiveness’ of differ- 
ent educational practices in school or class organization in 
relation to both cognitive and non-cognitive outcomes. These 

outcomes were also related to social, economic and ecological 
.,1 

factors,’ 

He points out that the important thing: 

*> £ n carrying out cross-national evaluations , is to cover 

the broadest possible spectrum of objectives aim contents at 

2 

a particular age or grade level* 

Husen emphasizes that the study did not seek to make compan- 
sons between countries as if in a contest. 



In the second phase of the I.E.A. Project ten-year-olds and 
fourteen-year-olds will be studied on achievement in science, mother 
tongue, civics, and a foreign language, in 19 countries. In addition 
to seeking to relate multiple variables t<j school achievement, this 
phase will attempt to evaluate system retention as a form of relative 

productivity. 

While Tyler notes the potential of cross-national evaluation, 
he agrees that the: 

"most critical theoretical problem is the definition of com- 
parability in terms of objectives, content, and target popu- 
„3 

lations . . . 

In an interview with Husen, we asked about the interest of 
I.E.A. in studies of a longitudinal nature, which would test the same 
sample of students at least a second time. The results might provide 



1 T. Husen , "International Impact of Evaluation", in Ralph W. Tyler 
(editor), op.cit . , pp • 340-341. 

2 Ibid. , p. 343. 

3 Ralph W. Tyler, op.cit . , p. 397. 
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further meaningful information about system, effectiveness. The reply 
was that this proposal was not at present in the plans of I.E.A. 

While such cross-national studies may not be forthcoming in the near 
future, we look to the development of longitudinal testing within 
the individual countries- 

On the national scale, numerous countries were not reported 
in the literature as conducting studies, nor was there material 
available to indicate that studies of output evaluation are of cur- 
rent interest or pre-occupation. In several countries, officials 
who were sent letters of inquiry about system evaluation, student 
achievement testing, or curriculum evaluation, failed to respond. 
Therefore, the absence of particular countries in the following re- 
view indicates our own inability to obtain information of consequence 

to this study. 

For England and Wales, a major research centre is the 
National Foundation for Educational Research. While many of the re- 
search projects of the NFER are concerned in some way with the evalua- 
tion of educational effectiveness, the greatest emphasis is now placed 
upon the collaboration with I.E.A. for the coming phase of testing 
in England. Thus, the ITFER is involved in preparation of tests and 
in the necessary planning for data collection in both the primary 

and secondary stages ot this project* 

Studies in England of pupil progress as related ^to motivation, 

expectation and home factors were reported by Pidgeon. Addi.ionol 
interest is shown in the effects of separating pupils for placement 
in the modern schools and in the grammar schools. Considered as 
having effects upon achievement. are such curricular and structural 
elements as grade placement versus age grouping and streaming, age 
of beginning school, teacher training, and differing social conditions 



1 Douglas A. Pidgeon, Expectation and Pupil Perfo rmance, Stockholm 
Alnqvist and Wiksell, 1970- 
Ibid., pp. 82-85. 
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As reported by Wall 1 , the I1FER has in the past conducted re- 
search studies on curriculum evaluation. Projects of the Nuffield 
Foundation have been studied as have earlier curricular programmes 
in reading and mathematics. The 1TFER for many years has provided 
test services for diagnosis of educational difficulties, day-to-day 
evaluation of pupils’ work, and for educational guidance. 

The Examinations and Tests Research Unit of the NFER studies 
the established standards and practices of C.S.E. examining boards 
to ensure comparability. This Unit is also undertaking research on 
secondary school pupils’ activities after having taken the C.S.E, 

Research reported in Wales notes a critical analysis of pre 
vious research on the effects upon academic attainment of co-educa- 
tion* This study was especially concerned with mathematic & and 
English* Other studies were of bilingualism (Welsh-Lnglish) and 
educational achievement, a problem which is relevant to both developed 
and developing countries. 

Among studies reported which reflect sociological aspects, 

are several which conclude that: 

"the general improvement in academic achievement in relation 
to ability in the post-war period has not bean evenly dis- 
tributed. The improvement has been greater xn the upper and 
middle classes and the reduction in the differential rate of 
achievement between the social classes has been small. This 
suggests that there is a continuing and ... even accelerating 
inadequacy in the British system of education in relation to 



1 W.D. Hall? "The Work of the National Foundation for Educational 

Research in England and Wales", in H.J. Butcher (editor) , Educational 
Research in Britain , London: University of London Press, 1968. 

2 Schools Council, Welsh Committee* Educational Res earch in Wales., 
H.M.S.O. , 1968. 
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its generally accepted selective function and that it is not 
an efficient agent for the sorting out of available ability 
in all classes.' 

The Scottish Council for Research in Education has undertaken 
surveys of scholastic achievement and has^participated in international 

comparative studies of pupil achievement. 

In his summary of curriculum evaluation research in Britain, 

Williams refers to important research strategies. 

(a) the taking into account of the special outcomes expected 
of each curriculum as v;ell as the over-all common out- 
comes and the need to formulate such expectations clearly; 
(b> the use of testing instruments that till have implications 
for the pupil’s performance later in his career. 

(c) the focus on short-term studies on the components of the 
curriculum rather than to attempt to treat the curricu- 
lum as a whole in evaluation; 

(d) evaluation of curricula should take place on a recurrent 
basis so as to determine patterns of effects; 

(e) the distribution of testing should make it possible to 
assess a variety of aspects of curriculum effectiveness 
by representative and equivalent pupil .samples. 



J.L. Williams, ’’Sociology and education in contemporary Wales , 
in Schools Council, op.cit ., pp . 42-43. In this report, Williams 
refers to the contributions of English researchers in sociology 
and education. 

D.A. Walker, "The Work of the Scottish Council for Research in 
Education", in K.J. Butcher (editor), QP-cit. , pp. 33-44. 

John Williams, ’’The Curriculum: Some Patterns of Development 

and Designs for Evaluation", in H.J. Butcher (editor), op.cit .. 
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In general, the above discussion is a relatively clear picture 
of the research on system evaluation now prevalent u Britain. Examina 
tions are monitored for standardization individual subject curricula 
are studied as to student achievement and with some follow-up studies, 
and international comparative studies continue. 

However, the problems continue to be given serious considera- 
tion as witnessed by the following; 

’’There is a need for a look at the whole exam system. The 
Schools Council has just set up a committee to look into the 
whole curriculum and a similar committee is needed on exams. 
Then the questions might be raised about where we need predic- 
tive tests, and where we need attainment tests, ... and what 
other forms of assessment need to be built in (teachers' re- 
ports, course work, and so on). 

"Now is a good time. The Council's curriculum projects are 

7? 1 

a stimulus to look at new forms of evaluation. 

It may be the task of the educational sociologist (or his 
reward) to engage in the more challenging considerations and research 

which were noted by Williams ( supra) . 

With regard to research on evaluation in France, we have noted 
already ( supra) that reforms are being undertaken in various aspects 
of the educational systan, but the examination system remains rela- 
tively unchanged. There is, however , a growing tendency to rely upon 
teacher judgment for the evaluation of student achievement. 

Research on marking procedures, therefore is being undertaken 
as reported by F . Bacher in the World Yearbook of Educat ion . " She 
notes the need to study methods for improving objectivity of teacher 
evaluations by marks. She concludes that the aims of pupils' educa- 
tion must first be defined, and only then can an attempt be made: 
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1 Ann e Corbett, "The Story of Q and F ,: , New Society, 2 July l970 » 

pp. 16-18. 

^ op *ci t . , pp. 95-100. 
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"to apply techniques designed to show whether they have or 
have not been attained. This is a field of research which 
has been little explored as yet.' 

Interviews with M. Reuchlin (Director of the Institut. nationale 
d’etude du travail et d' orientation prof essionnelle) and with 
J. Drevillon at the University of Caen indicated that research activi- 
ties in France are directed at improved and expanded efforts in stu- 
dent orientation and guidance. This is considered even more critical 
than before in view of increased enrolments in secondary education, 
the growing involvement and interest of students and parents in de- 
cisions about their continued schooling, and the greater reliance 
upon student dossiers for such guidance. 

French researchers, including M. Reuchlin, are also partici- 
pating in the I.E.A. studies reported earlier. 

An instance of the growing concern for further research is 

the statement of J* Majaulti 

"... thought precedes action. Particularly as regards edu- 
cation. 

"Educational research is relevant to the contents of instruc- 
tion, to methodology, and to the means of carrying it out. 

That is to say that all the problems are to be considered! 
j~ including_T studies of the reaction of students to differ- 
ent procedures followed in the various subjects and to the 

2 

results achieved . u 

While the above does not describe on-going or planned research, 
it is a call for the undertaking of research and the exposure of 

numerous problems to careful and objective study. 



1 Ibid,, p. 100. 

2 J. Majault , La revolution de 1 « enseignement , Paris: Robert Laffort 



1967, pp. 181-184. 



